Methods and systems of extended reality environment interaction based on eye motions

ABSTRACT

Systems and methods are described for extended reality environment interaction. An extended reality environment including an object is generated for display, and an eye motion is detected. Based on the detecting, it is determined whether the object is in a field of view for at least a predetermined period of time, and in response to determining that the object is in the field of view for at least the predetermined period of time, one or more items related to the object are generated for display in the extended virtual reality environment.

BACKGROUND

This disclosure relates to improved extended reality environmentinteraction and, in particular, systems and methods are disclosed fordetecting eye motion and performing operations in an extended realityenvironment reality environment based on the detected eye motion.

SUMMARY

Advancements in media technology have led to development of extendedreality (XR) technologies, such as virtual reality (VR), augmentedreality (AR) and mixed reality (MR) technologies. VR systems may fullyimmerse (e.g., giving the user a sense of being in an environment) orpartially immerse (e.g., giving the user the sense of looking at anenvironment) users in a three-dimensional, computer-generatedenvironment. The environment may include objects or items that the usercan interact with. AR systems may provide a modified version of reality,such as enhanced information overlaid over real world objects. MRsystems map interactive virtual objects to the real world. Such systemsmay utilize wearables, such as a head-mounted device, comprising astereoscopic display, or smart glasses.

XR systems introduce many challenges. For example, it may be difficultfor XR systems to detect when a user alters his or her field of view orfocus in the XR environment, since the wearable device being used toview the environment may not include an external device (e.g., a lens).As another example, although pupil dilation and constriction may varydepending on what a user is viewing in an XR environment or an amount oflight entering the eye of the user, a user may not have control over hisor her pupil, and thus monitoring the user's pupil may not be a reliableway to determine a gaze or field of view of the user within an XRenvironment. Worse, even if a field of view of the user is accuratelyascertained, if there are multiple objects in the field of view of theuser, it may be difficult to determine which object the user desires tointeract with.

In addition, current approaches to XR suffer from certain drawbacks. Inone approach, a user employs hand gestures or a joystick to navigate anXR environment. However, requiring such user inputs to interact with theXR environment may be cumbersome or inconvenient for the user, not tomention take away from the experience of XR (i.e., remind the user thatthe XR environment is not real). In addition, in current approaches toXR, it may not be possible for a user to conveniently obtain informationconcerning objects in his or her field of view or that he or sheinteracts with in the XR environment.

To overcome these problems, systems and methods are provided herein foridentifying an object in a field of view of a user, detecting eyelidmotion of the user, and based on such detection, regenerating fordisplay the object in an extended reality environment with a modifiedlevel of detail. Systems and methods described herein also providematching a detected eyelid motion and a stored eyelid motion identifierand performing an action on an object based on such matching. Inaddition, systems and methods are provided to generate an indicator toreflect a gaze shift of a user to a new portion of an extended realityenvironment including an object, and execute an action when a voicecommand is received while the indicator is in a vicinity of the object.Systems and methods described herein also provide for generating fordisplay within an extended reality environment opacity-based indicatorsin a vicinity of a portion of the extended reality environment includingan object, and varying opacity of such indicators based on an identifiedboundary of the object. In addition, systems and methods are provided toenable a user to conveniently obtain additional information about itemsin the extended reality environment.

In some aspects of the disclosure, the extended reality system generatesfor display an extended reality environment including a first object andreceives input from one or more sensors. Based on the received input,the system identifies the first object in a field of view and detects aneyelid motion, and in response to detecting the eyelid motion,regenerates for display the first object with a modified level ofdetail. Thus, eyelid motion can be monitored in order to overcomechallenges associated with determining which object in a field of viewof the user is of interest to the user. In addition, detecting sucheyelid motion of the user enables the user to view, for example, finerdetails of an object that appears to be far away from the user withinthe extended reality environment, which may improve the user experiencein the extended reality system, particularly for a user having impairedvision.

The extended reality environment may comprise a plurality of objectsincluding the first object and a second object in the field of view, andthe system may regenerate for display the first object with the modifiedlevel of detail in response to determining that the detected eyelidmotion is associated with the first object. If the system determinesthat the detected eyelid motion is associated with the second object,the system may regenerate for display the second object with a modifiedlevel of detail. The first object may be in one of a foreground or abackground in the field of view in the extended reality environment, andthe second object may be in the other of the foreground or thebackground in the field of view in the extended reality environment.

In some embodiments, regenerating for display the first object with themodified level of detail comprises presenting the object in a higherresolution. Additionally or alternatively, one or more actions may beperformed on the first object based on one or more detected eyelidmotions.

In some aspects of this disclosure, the system computes respectivevirtual distances of the plurality of objects with respect to a user,and identifying the first object in the field of view comprisesdetermining the first object is at a closest virtual distance to theuser of the respective virtual distances.

In some embodiments, detecting the eyelid motion comprises determiningan amount of motion of the eyelid and/or detecting the eyelid motioncomprises determining one or more eyelid levels. The system may detectthat a user is navigating from a first position to a new position in theextended reality environment, while the first object remains in thefield of view, and generate for display an updated version of the firstobject based on a perspective of the user at the new position.

In some aspects of the disclosure, an extended reality system generatesfor display an extended reality environment including an object, andstores in memory a table of eyelid motion identifiers and correspondingactions performable on the object in the extended reality environment.Using a sensor, the system detects an eyelid motion, and matches thedetected eyelid motion to one of the stored eyelid motion identifiers.In response to matching the detected eyelid motion to one of the storedeyelid motion identifiers, the system generates for display an updatedversion of the extended reality environment based on the action thatcorresponds to the matched eyelid motion. Thus, eyelid motion can bemonitored in order to overcome challenges associated with determiningwhich object is a field of view the user desires to interact with. Inaddition, detecting such eyelid motion of the user enables the user tointeract with an object that appears to be far away from the user withinthe extended reality environment, which may improve the user experiencein the extended reality system, particularly for a user having impairedvision.

The object may be selected from a plurality of objects in the extendedreality environment by detecting that a gaze of a user is directed atthe object. The system may generate for display a subset of the eyelidmotion identifiers performable on the object at which the gaze of theuser is directed (e.g., to remind or guide the user as to an action thata certain eyelid motion causes to be performed). The action of theplurality of actions may correspond to manipulating the object and/oraltering the appearance of the object (e.g., if the object is a book,the action may be flipping pages of the book, tilting the book, tearingout a page of the book, etc.). The system may detect that the user isnavigating from a first position to a new position in the extendedreality environment, while the gaze of the user remains on the object,and generate for display an updated version of the first object based ona perspective of the user at the new position, the updated version ofthe object having the altered appearance.

In some embodiments, a user may be associated with a user profilespecifying relationships between eyelid motion identifiers andcorresponding actions performable on the object in the extended realityenvironment. The actions performable on the object may vary based on atype of the object. To detect the eyelid motion the system may determinewhether the eyelid remains closed for a predetermined period of time,and match the detected eyelid motion to one of the stored eyelid motionidentifiers in response to determining that the eyelid remains closedfor the predetermined period of time (e.g., to ensure the eyelid motionis not involuntary blinking).

In some aspects of the disclosure, the extended reality system generatesfor display an extended reality environment comprising an object, anddetects, using a first sensor, that a gaze has shifted from a firstportion of the extended reality environment to a second portion of theextended reality environment, where the object is excluded from thefirst portion of the extended reality environment and included in thesecond portion of the extended reality environment. In response todetecting the gaze shift, the system generates for display within theextended reality environment an indicator of the shift in the gaze, anddetects, by using a second sensor, a voice command while the indicatoris in a vicinity of the object. In response to detecting the voicecommand, the extended reality system executes an action corresponding tothe voice command. Thus, extended reality may be leveraged incombination with voice to improve the user experience. Morespecifically, a user may conveniently use his or her eyes to navigate anextended reality environment (e.g., as a proxy for how a mouse ortrackpad is used with a desktop, laptop or mobile device), receivereal-time confirmation as to a location of his or her gaze, and performdesired actions in the environment via a voice command when an indicatorof the gaze of the user is in the vicinity of an object of interest inthe extended reality environment.

An interactive media guide may be provided on the display, and theabove-mentioned action may be an instruction related to a media assetaccessible via the interactive media guide. The voice command mayinclude an identification of the media asset and a command to executethe action, and/or an instruction to present a new media asset on thedisplay and/or an instruction to retrieve content related to an entity,where the object is associated with the entity.

In some embodiments, the extended reality system may determine whether arate of retinal movement exceeds a predetermined value, and in responseto determining that the rate of retinal movement exceeds thepredetermined value, normalize the retinal movement when translating theretinal movement into movement of the indicator on the display. Thesystem may detect the voice command while the indicator is in thevicinity of the object (e.g., overlapping the object) upon determiningthe gaze is directed at the object for at least a predeterminedthreshold period of time. The display is presented via a virtual realityhead-mounted device.

In some aspects of the disclosure, an extended reality system maygenerate for display an extended reality environment comprising anobject, and detect, by using a sensor, a gaze is directed to a firstportion of the extended reality environment, where the object isincluded in the first portion of the extended reality environment. Theextended reality system may generate for display within the extendedreality environment a plurality of opacity-based indicators in thevicinity of the first portion of the extended reality environment,identify a boundary of the object, and varying an opacity of the atleast one of the plurality of opacity-based indicators based on theidentified boundary of the object. Thus, a user may conveniently use hisor her eyes to navigate an extended reality environment (e.g., as aproxy for how a mouse or trackpad is used with a desktop, laptop ormobile device) and receive real-time confirmation as to a location ofhis or her gaze, where opacity of indicators of such real-time gaze areconveniently adjusted so as not to obscure the view of the user andavoid degrading the user's experience.

The extended reality system may determine whether the at least one ofthe opacity-based indicators overlaps the boundary of the object, andvary respective opacities of opacity-based indicators that overlap theboundary. The plurality of opacity-based indicators are arrows directedtowards the object. The extended reality system may detect, by using thesensor, whether the gaze has shifted to a second portion of the extendedreality environment, and in response to determining that the gaze hasshifted to the second portion, cause the plurality of opacity-basedindicators to be overlaid in a vicinity of the second portion of thedisplay.

In some embodiments, the respective opacities are varied based on adistance from the object. For example, the respective opacities of theindicators may increase as the distance between the indicator and theobject decreases (e.g., to emphasize the object the user is gazing at)or increase as the distance between the indicator and the objectdecreases (e.g., to avoid obscuring the object the user is gazing at).

In some embodiments, an interactive media guide may be is provided onthe display, and an action related to a media asset accessible via theinteractive media guide is received at least in part based on thedetected gaze. Such display may be presented via a virtual realityhead-mounted device or presented without the use of a virtual realityhead-mounted device.

In some aspects of the disclosure, an extended reality system generatesfor display a extended reality environment including an object, detectsan eye motion, and determines, based on the detecting, whether an objectis in a field of view for at least a predetermined period of time. Inresponse to determining that the object is in the field of view for atleast the predetermined period of time, the system generates for displayin the extended reality environment one or more items related to theobject. Thus, information for an object of interest may be convenientlydisplayed to the user based on detecting his or her eye motion relatedto the object of interest.

The one or more items related to the object may comprise textualinformation, images, video, or any combination thereof. The system mayfurther determine that at least a second predetermined period of timehas elapsed from commencing the display of the one or more items withoutthe object being in the field of view for the first predetermined periodof time, and cease display of the one or more items in response to suchdetermination. The extended reality environment may be presented via avirtual reality head-mounted device. In some embodiments, detecting theeye motion comprises monitoring an eyelid motion or monitoring gaze.

The system may determine whether the object is in the field of view forthe predetermined period of time upon determining that the field of viewis continuously (or non-continuously) on the object for thepredetermined period of time during a virtual reality session.

In some embodiments, the system may determine a new object is in thefield of view for at least the predetermined time, and in response tosuch determination, generate for display in the extended realityenvironment one or more items related to the new object, whilecontinuing to generate for display in the extended reality environmentthe one or more items related to the object.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and advantages of the present disclosurewill be apparent upon consideration of the following detaileddescription, taken in conjunction with the accompanying drawings, inwhich like reference characters refer to like parts throughout, and inwhich:

FIG. 1 shows an example of regenerating for display an object in anextended reality environment, in accordance with some embodiments ofthis disclosure;

FIG. 2 shows an example of regenerating for display an object in anextended reality environment, in accordance with some embodiments ofthis disclosure;

FIG. 3 shows an example of performing an action on an object in anextended reality environment, in accordance with some embodiments ofthis disclosure;

FIGS. 4A-4B show an example of receiving a voice command while anindicator is in a vicinity of an object in an extended realityenvironment, in accordance with some embodiments of this disclosure;

FIG. 5 shows an example of receiving a voice command while an indicatoris in a vicinity of an object in an extended reality environment, inaccordance with some embodiments of this disclosure;

FIG. 6 shows an example of receiving a voice command while an indicatoris in a vicinity of an object in an extended reality environment, inaccordance with some embodiments of this disclosure;

FIG. 7 shows an example of presenting information related to an item inan extended reality environment, in accordance with some embodiments ofthis disclosure;

FIG. 8 is a block diagram of an illustrative device in an extendedreality system, in accordance with some embodiments of this disclosure;

FIG. 9 is a block diagram of an illustrative extended reality system, inaccordance with some embodiments of this disclosure;

FIG. 10 is a flowchart of a detailed illustrative process forregenerating for display an object in an extended reality environment,in accordance with some embodiments of this disclosure;

FIG. 11 is a flowchart of a detailed illustrative process forregenerating for display an object in an extended reality environment,in accordance with some embodiments of this disclosure;

FIG. 12 is a flowchart of a detailed illustrative process for performingan action on an object in an extended reality environment, in accordancewith some embodiments of this disclosure;

FIG. 13 is a flowchart of a detailed illustrative process for receivinga voice command while an indicator is in a vicinity of an object in anextended reality environment, in accordance with some embodiments ofthis disclosure;

FIG. 14 is a flowchart of a detailed illustrative process for varying anopacity of indicators in an extended reality environment, in accordancewith some embodiments of this disclosure; and

FIG. 15 is a flowchart of a detailed illustrative process for presentingadditional information related to an item in an extended realityenvironment, in accordance with some embodiments of this disclosure.

DETAILED DESCRIPTION

FIG. 1 illustrates an exemplary process of regenerating for display anobject in an extended reality (XR) environment, in accordance with someembodiments of this disclosure. Head-mounted display 102 may projectimages to generate a three-dimensional XR environment 100 for immersinga user therein. The user may be fully or partially immersed in XRenvironment 100, and such environment may be a completely virtualenvironment. Head-mounted display 102 may alternatively be a wearabledevice (e.g., smart glasses), or a computer or mobile device equippedwith a camera and XR application, to facilitate generation ofenvironment 100. Environment 100 may alternatively be an augmentedreality (AR) environment in which real-world objects are supplementedwith computer-generated objects or information, or mixed reality (MR),e.g., where virtual objects interact with the real world or the realworld is otherwise connected to virtual objects. In some embodiments, aview or perspective of the user of environment 100 changes as the usermoves his or her head, and other features (e.g., audio) are suitablymodified, simulating the physical world. Environment 100 may be forentertainment purposes (e.g., video games, movies, videos, sports,etc.), communication (e.g., social media), educational purposes (e.g., avirtual classroom), professional purposes (e.g., training simulations),medical purposes, etc.

The XR system may identify one or more objects in a field of view of theuser. A field of view is a portion of XR environment 100 that ispresented to the user at a given time by the display 102 (e.g., an anglein a 360-degree sphere environment). The field of view may comprise apair of 2D images to create a stereoscopic view in the case of a VRdevice; in the case of an AR device (e.g., smart glasses), the field ofview may comprise 3D or 2D images, which may include a mix of realobjects and virtual objects overlaid on top of the real objects usingthe AR device (e.g., for smart glasses, a picture captured with a cameraand content added by the smart glasses). If an XR environment has asingle degree of liberty, e.g., a rotation of 360 degrees, any field ofview may be defined by either the edge angular coordinates (e.g., +135degrees, +225 degrees) or by a single angular coordinate (e.g., −55degrees) combined with the known angular opening of the field of view.If an XR environment has six degrees of liberty, say three rotations of360 degrees and three spatial positions, any field of view may bedefined by three angular coordinates and three spatial coordinates. Afield of view may therefore be understood as a portion of the XRenvironment displayed when the user is at a particular location in theXR environment and has oriented the XR set in a particular direction.

An XR system (e.g., system 900 of FIG. 9) may generate a data structurefor the field of view, including object identifiers associated withvirtual objects in the field of view, and such data structure mayinclude coordinates representing the position of the field of view inthe XR environment. The system may determine the present field of viewbased on the data structure and/or images captured by the XR device 102,and identify objects in the field of view of the user. As shown in theexample of FIG. 1, the detected field of view of the user in environment100 includes object 104, depicted as a car, although one of skill in theart will appreciate that any number or combination of different types ofobjects may be included in environment 100. The XR system may generatefor display object 104 in a default level of detail (e.g., a defaultresolution or number of displayed pixels, or a default size orappearance). For example, objects in environment 100 may be presented bydefault in 4K resolution (3840×2160), or any other suitable resolution.The resolution of objects in environment 100 may be the same, or vary,for each eye of the user. In some embodiments, the level of detail mayrefer to a size or appearance of the object, e.g., the object may begenerated at a default size or default color.

In some embodiments, upon determining the one or more objects 104 in thefield of view of the user, the XR system may generate for displayidentifiers 106 (e.g., “Blink once to modify details of car”), which mayindicate or otherwise provide guidance to the user as to how aparticular eyelid motion causes certain actions to be performed onobject 104. In some embodiments, the XR system may reference a table(e.g., table 316 of FIG. 3, which may be stored in storage 808 of FIG.8) that includes a plurality of eyelid motion identifiers andcorresponding actions performable on object 104 in the XR environment.For example, the table may additionally store an identifier (e.g.,blinking twice) which may correspond to increasing or decreasing thesize of the object displayed to the user upon detecting the indicatedeyelid motion.

Once the objects of interest in the field of view are identified, the XRsystem may detect an eyelid motion 108 of the user by using a sensor(e.g., a camera). In some embodiments, the XR system may detect whethereyelid motion exceeds a predetermined period of time (e.g., 0.5 secondsor 1 second) in order to avoid performing an action based on aninvoluntary blink (e.g., if such action is not desired by the user). Inresponse to detecting the eyelid motion 108 of the user (e.g., a singleblink corresponding to an action of modifying details of the object ofinterest), the XR regenerates for display object 104, provided to theuser via head-mounted display 102. For example, object 104 may bepresented to the user at a higher resolution (e.g., 8K resolution,7680×4320) than initially provided (e.g., 4K resolution, 3840×2160). Insome embodiments, the detected eyelid motion may cause the XR system tomodify details of object 104 in a different manner (e.g., increasing ordecreasing the size of the object as compared to the initialpresentation of the object, changing the color or texture of an objectas compared to an initial appearance of the object, etc.).

In some embodiments, detecting the eyelid motion comprises determiningan amount of motion of the eyelids or detecting the eyelid motioncomprises determining one or more eyelid levels. For example, the XRsystem may detect an amount of motion of the eyelids using a sensor(e.g., a camera) and may compare the detected an amount to a thresholdamount of motion over a predetermined period of time (e.g., five eyelidmotions detected over a three-second period of time), and an image maybe modified or selected when the detected amount of motion of theeyelids exceeds the threshold amount of motion over the predeterminedperiod of time. As another example, the XR system may detect, using asensor (e.g., a camera), one or more eyelid levels (e.g., distincteyelid levels) over a predetermined period of time, and may compare thedetected amount to a threshold number of eyelid levels over apredetermined period of time (e.g., five distinct eyelid levels detectedover a three second period of time), and an image may be modified orselected when the detected number of eyelid levels exceeds the thresholdnumber of eyelid levels over the predetermined period of time.

FIG. 2 shows an exemplary process in which multiple objects are detectedin the field of view of the user. The XR system may detect that objects204 and 208 are each in the field of view of the user being displayed XRenvironment 200 via head-mounted display 202. As shown in environment200 at a top-most portion of FIG. 2, the objects 204, 208 detected asbeing in the field of view of the user may initially be presented with adefault level of detail (e.g., a default resolution, or a default size).Upon detecting eyelid motion 210 (e.g., corresponding to “Blink once tomodify details of car” indicated in identifiers 206), the XR system mayregenerate for display object 204 in the field of view with a modifiedlevel of detail (e.g., enhance the resolution of object 204). On theother hand, upon detecting eyelid motion 212 (e.g., corresponding to“Blink twice to modify details of airplane” indicated in identifiers206) the XR system may regenerate for display object 208 in the field ofview with a modified level of detail (e.g., enhance the resolution ofobject 208).

In some embodiments, detecting further eyelid motion 212 may cause themodification performed in response to detecting eyelid motion 210 tosubsequently be reversed (e.g., object 204 may revert to the defaultresolution initially presented to the user, while object 208 ispresented with modified details). Alternatively, an object may bemaintained in the modified state throughout the XR session, and/or infuture sessions. In some embodiments, detecting that an eyelid motion isre-performed may cause the action to be reversed (e.g., detecting eyelidmotion 210 a second time may cause object 204 to revert to the defaultresolution initially presented to the user). In some embodiments, of theplurality of objects that may be in the field of view of the user, oneof such objects (e.g., object 204) may be in the foreground of thedisplay of XR environment 200, and the other of such objects may be inthe background of the display (e.g., object 208) in XR environment 200.In addition, one or more actions may be performed on the modified objectin the field of view of the user (e.g., a particular eyelid motion maycorrespond to opening the door of the car object 204, or interactingwith the airplane object 208). The most recently modified object in thefield of view may be a “selected” object such that actions may beperformed on such object. Such aspects may enable an object that isdistant from the user, or otherwise too small to be seen in detail, tobe regenerated for display in modified detail to allow the user toclearly view the object.

In some aspects of this disclosure, the XR system may detect the eyelidmotions of the user in congruence with the objects in his or her fieldof view, and may compute relative extent to which an eyelid is closed,to determine which object to initially focus on in the user's field ofview. In some embodiments, when the user enters the XR environment, theXR system may set a default field of view, detect the number of objectsin the environment and/or in the field of view, and compute respectivevirtual distances, or focal lengths, of the each of the detected objectswith respect to a user. The objects may be at different virtualdistances from the user. In some embodiments, identifying an object inthe field of view comprises determining the object is at a closestvirtual distance to the user of the respective virtual distances orfocal lengths. The virtual distance may be, for example, the perceiveddistance the object in the XR environment is located from the user, andmay be calculated based on coordinates of the object in the XRenvironment. Eyelid levels of the user may be calculated at least inpart based on such virtual distances, and upon detecting a change ineyelid level, an object that is a closest virtual distance to the usermay be detected and selected as the object of interest, to whichmodifications may be performed.

In some embodiments, the XR system may detect movement of the useraround XR environment 200, and that, as the user moves around, his orher field of view changes. In such circumstance, the field of view ofthe user may be reset, in order to determine a number of objects in theuser's new field of view. On the other hand, if the XR system detectsmovement of the user around XR environment 200, but that the gaze of theuser still remains fixed on a particular object, the display 200 maygenerate for display such object from varied perspectives, consistentwith the movement of the user, to maintain the simulated environment. Insome embodiments, any change of eyelid levels detected by the virtualreality system may be used to determine the object, the detail of whichis to be modified, in the field of view of the user. The XR system maytrack the user's movements within the XR environment by using sensors(e.g., gyroscopes, accelerometers, cameras, etc., in combination withcontrol circuitry).

FIG. 3 shows an exemplary process of performing an action on an objectin an XR environment, in accordance with some embodiments of thisdisclosure. Head-mounted display 302 may generate for display XRenvironment 300 including objects 304, 310, 312, 314. Although fourobjects are shown in environment 300 (book object 304, lamp object 310,desk object 312 and chair object 314), it should be appreciated that anynumber of objects, and any type of objects, may be generated fordisplay. The XR system (e.g., system 900 of FIG. 9) may store (e.g., instorage 808 of FIG. 8) table 316 of eyelid motion identifiers andcorresponding actions performable on the object in the XR environment.For example, table 316 may store a plurality of associations, includingassociations for the book object 304, as indicated by identifiers 306,which may be displayed to the user to facilitate desired actions: “Blinkonce to flip pages of book; Blink twice to tilt book; Blink three timesto tear page from book.” It will be appreciated that any number ofactions, and any type of identifier, may be stored in table 316, andthat an action may depend on the type of object (e.g., table 316 maystore an identifier associated with an action to turn on a virtual lightbulb in connection with lamp object 310). In a case where theenvironment includes multiple objects, an object may be selected fromthe plurality by detecting (e.g., using a sensor) that a gaze of a useris directed at the object.

The XR system detects, by using a sensor (e.g., a camera), an eyelidmotion of the user. The system may determine whether the detected eyelidmotion matches any of the identifiers in table 316, e.g., by analyzingthe sensor output and comparing such output to the stored identifiers(e.g., predetermined number of blinks, blink patterns, amount of eyelidmotion, eyelid levels, etc.). In some embodiments, the storedidentifiers may include eyelid motions in combination with voicecommands or other inputs. In some embodiments, the XR system may detectwhether eyelid motion exceeds a predetermined period of time (e.g., 0.5seconds) in order to avoid performing an action based on an involuntaryblink (e.g., if such action is not desired by the user). The system maydetect eyelid motion of the user based on an extent of opening andclosing of eyelids of the user over time.

In response to matching the detected eyelid motion to one of the storedeyelid motion identifiers, the XR system generates for display anupdated version of XR environment 300 based on the action thatcorresponds to the matched eyelid motion. In the example of FIG. 3, thesystem detects eyelid motion associated with flip page of book 318, andexecutes such action, as shown in environment 300 in the bottom portionof FIG. 3 depicting a flipped page of the book object 304, as comparedto the book object 304 in the top portion of the environment 300depicting a closed book. One of skill in the art will appreciate thatthe objects in environment 300 may be manipulated in various ways, e.g.,chair object 314 may be moved adjacent to a different portion of tableobject 312, or altered in various ways, e.g., removing a cushion fromchair object 314.

In some embodiments, a subset of the identifiers 306 suitable for aselected object of interest may be displayed to the user, for theconvenience of the user in determining available actions to be performedbased on a particular eyelid motion. In some embodiments, the XR systemmay store one or more user profiles specifying relationships betweeneyelid motion identifiers and corresponding actions performable on theobject in XR environment 300. For example, the user profile may include,e.g., actions tailored to the user preferences, favorite actions of theuser, most recently performed actions of the user, most commonlyperformed actions of the user, purchase actions of the user, etc., whichmay be displayed in association with identifiers 306 for the convenienceof the user.

In some embodiments, the XR system may detect movement of the useraround XR environment 200, and may detect, as the user moves around,that the gaze of the user changes. In such circumstance, the system mayselect a new object of interest. Alternatively, the system may detectthat the user is navigating from a first position to a new position inthe XR environment, while the gaze of the user remains on an object, andin response to such determination, generate for display an updatedversion of the object based on a perspective of the user (e.g., alterthe size or angle of the object presented to the user). The updatedversion of the object may include presenting the object to the userhaving the altered appearance (e.g., the book with a torn page, in theevent the user previously performed the eyelid motion associated withsuch action in table 316).

In some embodiments, the aspects discussed in FIGS. 1-2 may be combinedwith the embodiments of FIG. 3 (e.g., objects in the XR environment maybe regenerated in more detail, and various actions may be performed onsuch objects, in a single user session in the XR environment).

FIGS. 4A-4B show an example of receiving a voice command while anindicator is in a vicinity of an object in an XR environment, inaccordance with some embodiments of this disclosure. The XR system(e.g., system 900 of FIG. 9) may generate for display via head-mounteddisplay 402 XR environment 400 to the user. In some embodiments, XRenvironment 400 may include an interactive media guide application tofacilitate selection and consumption of media content. XR environment400 may include one or more objects 408, 410, which may correspond toidentifiers for selectable media content. The system detects by using asensor (e.g., a camera), that a gaze of the user has shifted from aportion of the XR environment (e.g., in the vicinity of object 408; FIG.4A) to another portion of the XR environment (e.g., in the vicinity ofobject 410; FIG. 4B). It should be appreciated that FIGS. 4A-4B areexemplary, and the gaze of the user has shifted from a portion of XRenvironment 400, which may contain no objects or multiple objects, toanother portion of XR environment 400.

In response to detecting the gaze shift, the XR system may generate fordisplay, within XR environment 400, indicator 406 indicating the shiftin the gaze. For example, indicator 406 in FIG. 4A reflects that thegaze of the user is on object 408 (e.g., an identifier for the movie“The Dark Knight), and in FIG. 4B indicator 406 reflects the gaze of theuser has shifted to object 410 (e.g., an identifier for the movie“American Psycho”). In some embodiments, a single indicator may begenerated for display, or alternatively multiple indicators may begenerated for display. In some embodiments, the indicators may vary intranslucence based on proximity to the object of interest. In theexample of FIGS. 4A-4B, indicator 406 is shown as arrows directed to theobject of interest, although it will be appreciated by those of skill inthe art that the indicator may comprise any suitable indicia or markingto cause the associated object to be emphasized or prominently displayedto the user. For example, the indicators may be a certain color or shapeto highlight the object of interest, images or emojis of eyeballs,magnification of the object in the vicinity of the indicators, animationof the object in the vicinity of the indicators, etc.

The system may detect, by using a sensor (e.g., a microphone), a voicecommand while indicator 406 is in a vicinity of object 410 (e.g., if theindicator overlaps, or otherwise is within a predetermined distance of,the object of interest). The XR system may process the voice command,and execute the action (e.g., provided there is a match between theobject included in the voice command and the object at which the gaze ofthe user is directed, as indicated by indicator 406). For example, uponreceiving the voice command 404 in the example of FIG. 4A, the systemmay commence presentation of the media asset “The Dark Knight”associated with object 408, and upon receiving voice command 405 in theexample of FIG. 4B, the system may commence presentation of the mediaasset “American Psycho” associated with object 410. In some embodiments,if it is determined by the user that the indicators are not accuratelyreflecting his or her gaze, the system may accept a suitable voicecommand from the user requesting the system to recalibrate his or hergaze, and/or indicating which portion of the display the user believeshe or she is gazing at.

In some embodiments, the gaze of the user is detected based on a retinalmovement of the eye (tracked by a sensor, e.g., a camera measuringreflections of a light source off the retina, eye tracking glasses,screen-based eye tracking). The retinal movement of the user may beplotted or translated to the display of the XR environment as movementof indicator 406 on the display. In some aspects of this disclosure, thesystem may determine whether a rate of retinal movement exceeds apredetermined value, and in response to such determination, performsnormalization when translating the retinal movement into movement ofindicator 406 on the display of the XR environment. For example, if thespeed of the gaze shift exceeds a predetermined threshold, normalizationmay be performed to slow movement of indicator 406 on the display (e.g.,to enable the user to more easily track the movement of indicator 406 onthe display). The entire cluster of indicators may move to such newportion of the display.

In some embodiments, the system includes an electronic voice recognition(or voice-assisted) device (e.g., a television, a computer, a voiceassistant) responsive to user voice commands, and the voice input may bein the form of audio or digital signals (or audio or digital input). Thesystem may perform natural language understanding (NLU) techniques, andmay include natural language understanding circuitry and/orspeech-to-text circuitry to transcribe the voice command to text, andmay parse the voice command to identify and extract keywords from thevoice input. The system may compare the extracted keyword to metadataassociated with an object of interest to determine whether there is amatch, e.g., whether to execute the voice command. In some embodiments,if the received voice command does not match the object in the vicinityof the indicator, the system may notify the user of the mismatch andrefrain from executing the associated action, or prompt the user for anew voice command.

In some embodiments, the voice command includes an identification of themedia asset and a command to execute the action (e.g., play,fast-forward, rewind, etc.), or an instruction to present a new mediaasset on the display (e.g., to scroll through other media assets or moveto a new page of media assets in a carousel). In some aspects of thisdisclosure, determining that the indicator is in the vicinity of theobject comprises determining that the gaze of the user is directed atthe object for at least a predetermined threshold period of time (e.g.,five seconds).

FIG. 5 shows an example of receiving a voice command while an indicatoris in a vicinity of an object in an XR environment, in accordance withsome embodiments of this disclosure. XR system (e.g., system 900 of FIG.9) may include head-mounted display 502 and generate for display XRenvironment 500 including objects 508, 510 by way of head-mounteddisplay 502. As shown in the top environment 500 of FIG. 5, the systemmay detect, by using a sensor (e.g., a camera), a gaze of the user isdirected to a portion of the XR environment (e.g., including object508). The system may generate for display within XR environment 500 aplurality of opacity-based indicators 506 in the vicinity of the portionincluding object 508 of XR environment 500. In the example of FIG. 5,indicator 506 is shown as arrows directed to the object of interest,although it will be appreciated by those of skill in the art that theindicator may comprise any suitable indicia or marking to cause theassociated object to be emphasized or prominently displayed to the user.For example, the indicators may be a certain color or shape to highlightthe object of interest, images or emojis of eyeballs, magnification ofthe object in the vicinity of the indicators, animation of the object inthe vicinity of the indicators, etc.

The system may identify boundaries (e.g., edges, shape outline, border)of object 508, e.g., by edge detection techniques, retrievingcoordinates of object 508, analyzing pixel values of the areasurrounding object 508, etc. Based on the identified boundary of theobject, the XR system may vary an opacity of at least one of theplurality of opacity-based indicators 506. In some embodiments, thesystem may determine whether at least one of the plurality ofopacity-based indicators overlaps, or is within a predetermined distanceof, the boundary of the object, and in response to such determination,may vary the respective opacities of the one or more indicators 506 thatoverlap the boundary of object 508. For example, the system may comparecoordinates of the object of interest in XR system 900 of FIG. 9 tocoordinates of the indicators 506. In some embodiments, if the systemdetects that the gaze of the user shifts from a portion of the display(e.g., including object 508) to a portion of the display includinganother object (e.g., object 510), the system causes the plurality ofopacity-based indicators to be overlaid in a vicinity of the portion ofthe display including object 510. The entire cluster of indicators maymove to such new portion of the display.

The system may vary the respective opacities based on a distance fromthe object. As shown in the example of FIG. 5, the respective opacitiesof indicators 506 may increase as the distance between indicator 506 andobject 508 decreases. This may be desirable in order to emphasize to theuser the portion of the display to which his or her gaze is directed.Alternatively, as shown in the example of FIG. 6, the respectiveopacities of indicators 606 may decrease as the distance between therespective indicator 606 and object 608 decreases. This may be desirablein order to minimize obscuring portions of the object of interest. Insome embodiments, the system may determine whether any of the indicatorsoverlap or are otherwise in a vicinity of another object, which may notbe of interest, and in such circumstance, the indicators may be set tobe translucent to avoid either obscuring portions of such object not ofinterest or incorrectly indicating to the user that such object is ofinterest.

As shown in the examples of FIG. 5 and FIG. 6, XR environment 500, 600,respectively, may include an interactive media guide. In response toreceiving a voice command to perform an action associated with an objectin the vicinity of indicators 506, 606, the system may perform theaction (e.g., generate for presentation identifiers for movies havingmetadata associated with “Christian Bale,” the actor associated withobject 508, 608). Although the examples of FIGS. 5-6 show objectsrelated to an “actor” category, various other categories may begenerated for display and browsed (e.g., genre, cast, director, etc.).In some embodiments, the environments 500, 600 may be presented with orwithout an XR head-mounted device. For example, the XR environment maybe presented as fully immersive VR, or in AR overlaid on a televisionscreen such as shown in FIGS. 5-6, or MR.

In some embodiments, the features of the examples of FIGS. 1-4 may beused in conjunction with the features of the examples of FIGS. 5-6. Forexample, indicators (e.g., such as indicators 106 in FIG. 1) may bedisplayed to the user to guide the user in performing eyelid motions orgazes, which may be associated with performable actions (e.g., in table316 of FIG. 3). The indicators of FIGS. 5 and 6 may also be used inconnection with other environments (e.g., video games, sports,communications, social media, etc.).

FIG. 7 shows an example of presenting additional information related toan item in an XR environment, in accordance with some embodiments ofthis disclosure. Head-mounted display 702 presents XR environment 700 tothe user. XR environment 700 may include one or more objects, e.g.,washing machine object 702, refrigerator object 704, wall object 708,etc. In some embodiments, XR environment 700 includes identifier 706,which may indicate to the user actions that may be taken in environment700. Although environment 700 shows a room in a building (e.g., a houseor apartment), environment 700 may comprise any virtual or augmentedenvironment, e.g., a mountain range, a scenic location, a video gamebeing played by the user, etc.

The system detects an eye motion of the user (e.g., via a sensor), andbased on the detected eye motion, determines whether an object (e.g.,washing machine object 702, refrigerator object 704, and/or wall object708) is in a field of view of the user for at least a predeterminedperiod of time 710 (e.g., five seconds). In response to determining thatan object (e.g., wall object) is in the field of view for at least thepredetermined period of time, the system generates for display in the XRenvironment one or more items 714 related to object 708. One or moreitems 714 related to the object may comprise textual information,images, video, or any combination thereof, and may be retrieved fromitem information database 712 (e.g., stored in storage 808 of FIG. 8).For example, item 714 indicates a thickness of wall object 708, and suchadditional information may be provided when a user gazes at wall object708 via head-mounted display 702 in XR environment 700. In someembodiments, detecting the eye motion comprises monitoring an eyelidmotion. In some aspects of the disclosure, generating for display in theXR environment one or more items 714 related to object 708 is performedin response to detecting both that object 708 is in the field of theview, as well as in a gaze of the user, for at least the predeterminedperiod of time. In some embodiments, the system permits modification ofthe predetermined period of time (e.g., by a user, or recommended by thesystem based on historic data of the user).

As another example, environment 700 may be a mountain range, and upondetermining, based on detected eye motion (e.g., detected via a sensor)of the user, that an object (e.g., a mountain) is in the field of viewof the user for at least a predetermined period of time, informationconcerning the mountain object may be generated for display to the user(e.g., the name of the mountain, the elevation of the mountain, thelocation of the mountain, etc.).

In some aspects of this disclosure, determining whether the object is inthe field of view for the predetermined period of time may comprisedetermining that the field of view is continuously on the object for thepredetermined period of time (e.g., five seconds) during an XR session.For example, if the system detects that the field of view of the userhas shifted during countdown of a timer to the predetermined period oftime, the countdown may be reset.

Alternatively, determining whether the object is in the field of viewfor the predetermined period of time may comprise determining that thefield of view is non-continuously on the object for the predeterminedperiod of time (e.g., five seconds) during an XR session. For example,if the system detects that the field of view of the user has shiftedduring countdown of a timer to the predetermined period of time, thecountdown may be paused, and resumed upon detecting that the gaze of theuser has returned to the object.

In some embodiments, the system may determine whether a new object is inthe field of view for at least the predetermined time. For example, inresponse to the system making such a determination, the system maygenerate for display in the XR environment one or more items related tothe new object, while continuing to generate for display in the XRenvironment the one or more items related to the object (e.g., object708 and item 714). This enables a user to view additional informationfor multiple items during at least the duration of the XR session. Insome embodiments, the additional information items may be maintained ina next user session in the XR environment.

In some embodiments, the system may determine that at least apredetermined period of time (e.g., 10 seconds) has elapsed fromcommencing the display of one or more items 714 without object 708 beingin the field of view for the first predetermined period of time. Forexample, if at least the predetermined period of time (e.g., 10 seconds)has passed since item 714 was provided to the user, and object 708associated with item 714 has not been in the field of view during suchperiod of time, display of item 714 may be ceased (e.g., since the usermay no longer be interested in object 708).

In certain embodiments, additional input (e.g., voice commands, handgestures, touch) may be utilized in combination with detecting the eyemotion of the user. For example, the system may receive a voice commandindicating that the user no longer wishes to view additional informationrelated to an item of interest, or may receive a voice command todisplay additional information, and may retrieve additional informationrelated to the item of interest (e.g., item 708) from item informationdatabase 712.

FIG. 8 is a block diagram of an illustrative device in an XR system, inaccordance with some embodiments of this disclosure. Device 800 in an XRsystem may include one or more servers for generating, displaying,and/or managing an XR environment, transmitted over a computer networkto XR devices (e.g., devices 102, 202, 302, 402, 502, 602, 702 of FIGS.1-7, respectively). The device 800 may include control circuitry 804 andan I/O path 810. Control circuitry 804 may be based on any suitableprocessing circuitry such as processing circuitry 806. As referred toherein, processing circuitry should be understood to mean circuitrybased on one or more microprocessors, microcontrollers, digital signalprocessors, programmable logic devices, field-programmable gate arrays(FPGAs), application-specific integrated circuits (ASICs), etc., and mayinclude a multi-core processor (e.g., dual-core, quadcore, hexa-core, orany suitable number of cores) or supercomputer.

In some embodiments, processing circuitry 806 may be distributed acrossmultiple separate processors or processing units, for example, multipleof the same type of processing units (e.g., two Intel Core i7processors) or multiple different processors (e.g., an Intel Core i5processor and an Intel Core i7 processor). I/O path 810 may providedevice information, or other data, over a local area network (LAN) orwide area network (WAN), and/or other content and data to controlcircuitry 804, which includes processing circuitry 806 and storage 808.Control circuitry 804 may be used to send and receive commands,requests, signals (digital and analog), and other suitable data usingI/O path 810. I/O path 810 may connect control circuitry 804 (andspecifically processing circuitry 806) to one or more communicationspaths. In some embodiments, storage 808 may include table 316 of FIG. 3and/or item information database 712 of FIG. 7.

In some embodiments, control circuitry 804 executes instructions forexecution of XR content generation and/or processing functions stored inmemory (e.g., storage 808). The instructions may be stored in either anon-volatile memory 814 and/or a volatile memory 812 and loaded intoprocessing circuitry 806 at the time of execution. A system forgenerating and processing XR content (e.g., the systems described inreference to FIGS. 1-7) may be a stand-alone application implemented ona user device (e.g., end-user device 920) and/or a server (e.g., server900) or distributed across multiple devices in accordance with device800. The system may be implemented as software or a set of executableinstructions. The instructions for performing any of the embodimentsdiscussed herein of XR processing may be encoded on non-transitorycomputer-readable media (e.g., a hard drive, random-access memory on aDRAM integrated circuit, read-only memory on a BLURAY disk, etc.) ortransitory computer-readable media (e.g., propagating signals carryingdata and/or instructions). For example, instructions in accordance withthe processes described herein may be stored in storage 808 and executedby control circuitry 804 of device 800.

Control circuitry 804 may include video-generating circuitry and tuningcircuitry, such as one or more analog tuners, one or more MPEG-2decoders or other digital decoding circuitry, high-definition tuners, orany other suitable tuning or video circuits or combinations of suchcircuits. Encoding circuitry (e.g., for converting over-the-air, analog,or digital signals to MPEG signals for storage) may also be included.Control circuitry 804 may also include scaler circuitry for upconvertingand downconverting content into the preferred output format of theend-user devices 920, 925, 930 of FIG. 9. Circuitry 804 may also includedigital-to-analog converter circuitry and analog-to-digital convertercircuitry for converting between digital and analog signals. The tuningand encoding circuitry may be used to receive and to display, to play,or to record content. The tuning and encoding circuitry may also be usedto receive guidance data. The circuitry described herein, including, forexample, the tuning, video generating, encoding, decoding, encrypting,decrypting, scaler, and analog/digital circuitry, may be implementedusing software running on one or more general purpose or specializedprocessors. Multiple tuners may be provided to handle simultaneoustuning functions (e.g., watch and record functions, functions forincorporating video or other recordings of physical environments,multiple-tuner recording, etc.). If storage 808 is provided as aseparate device from device 800, the tuning and encoding circuitry(including multiple tuners) may be associated with storage 808. Device800 may be a central device in communication with each device (e.g.,102, 202, 302, 402, 502, 602, 702). Alternatively, device 800 maycorrespond to device 104, that is, each system (e.g., of FIGS. 1-7) maynot have its own device 920, but rather a single central device 800 maycarry out XR environment generation and processes for each system.

FIG. 9 shows an illustrative block diagram of an XR content systemaccording to some embodiments of the disclosure. At least one XR contentserver 900 generates XR content, such as that described herein. XRcontent server 900 may transmit the content over internet 910 tomultiple end-user devices including devices 920, 925, 930. End-user XRdevices may include, for example, personal computers, mobile devices(e.g., smart phones), and/or wearable devices including XR headsets,goggles, suits, gloves, etc., configured to present and allowinteraction with XR environments. These devices are configured to allowan operator/user to view and interact with multi-user XR content (e.g.,via a display screen). These devices may provide, for example, visual,audio, and haptic feedback that presents the perspectives andattention-directing cues such as described herein. End-user devices 920,925, 930 also transmit data to server 900 via internet 910, where suchdata includes orientation information indicating the direction thedevices 920, 925, 930 are facing (and thus the view that server 900 mustgenerate for display on each device 920, 925, 930), audio signalsdetected by each device 920, 925, 930, and user input such as selectionsof XR objects. Server 900 may then generate a view of the XR environmentfor each device 920, 925, 930.

FIG. 10 is a flowchart of a detailed illustrative process forregenerating for display an object in an XR environment, in accordancewith some embodiments of this disclosure. It should be noted thatprocess 1000 or any step thereof could be performed on, or provided by,any of the devices shown in FIGS. 8-9. For example, process 1300 may beexecuted by network device 800 (e.g., via control circuitry 804), and/orcontrol circuitry of the server 900, and/or control circuitry of device920, 925, 930, as instructed by one or more programs ofcomputer-executable instructions that may be implemented on networkdevice 800, and/or server 900, and/or device 920, 925, 930, such as todistribute control of the extended reality environment among multipledevices. In addition, one or more steps of process 1000 may beincorporated into or combined with one or more steps of any otherprocess or embodiment (e.g., process 1100 of FIG. 11, process 1200 ofFIG. 12, process 1300 of FIG. 13, process 1400 of FIG. 14, process 1500of FIG. 15).

At 1002, input/output (I/O) circuitry (e.g., I/O circuitry ofhead-mounted display 102) may receive a user selection to enter an XRenvironment (e.g., a VR, AR, or MR environment). Such user selection maybe received via input (e.g., voice command or touchscreen).

At 1004, control circuitry (e.g., control circuitry 804 of FIG. 8 and/orcontrol circuitry of head-mounted display 102 of FIG. 1) may, inresponse to receiving the user selection, generate for display (e.g.,via head-mounted device 102 of FIG. 1) the XR environment including anobject (e.g., car object 104 of FIG. 1)

At 1006, I/O circuitry (e.g., I/O circuitry of head-mounted display 102)may receive input from a sensor (e.g., a camera). The sensor may detectvarious attributes of the eye of the user (e.g., eyelid motion, gaze).

At 1008, the control circuitry (e.g., control circuitry 804 of FIG. 8and/or control circuitry of head-mounted display 102 of FIG. 1) mayidentify an object (e.g., car object 104 of FIG. 1) in a field of viewof the user, based on the received input. In some embodiments, thecontrol circuitry may generate a data structure for the field of view,including object identifiers associated with virtual objects in thefield of view, and such data structure may include coordinatesrepresenting the position of the field of view in the XR environment.The control circuitry may determine the present field of view based onthe data structure and/or images captured by the XR device 102, andidentify objects in the field of view of the user.

At 1010, the control circuitry may detect eyelid motion (e.g., an amountof eyelid motion, a blink pattern, eyelid level, etc.) of the user basedon the received input from the sensor (e.g., a camera). In someembodiments, such eyelid motion may be distinguished from involuntaryblinks, to avoid interpreting such involuntary blinks of the user as acommand. For example, the system may disregard blink patterns where theeye remains closed for less than a predetermined amount of time (e.g.,0.5 seconds).

At 1012, the control circuitry may regenerate for display the object(e.g., car object 104 of FIG. 1) with a modified level of detail, inresponse to detecting the eyelid motion. For example, the controlcircuitry may regenerate for display the object in a higher resolutionupon detecting a particular blink pattern. In some embodiments, thecontrol circuitry may reference a table (e.g., table 316 of FIG. 3),that stores associations between eyelid motion identifiers andperformable actions, to determine whether the detected eyelid motionmatches the eyelid motion identifier for modifying a level of detail ofthe object.

FIG. 11 is a flowchart of a detailed illustrative process forregenerating for display an object in an XR environment, in accordancewith some embodiments of this disclosure. For example, process 1100 maybe executed by network device 800 (e.g., via control circuitry 804),and/or control circuitry of the server 900, and/or control circuitry ofdevice 920, 925, 930, as instructed by one or more programs ofcomputer-executable instructions that may be implemented on networkdevice 800, and/or server 900, and/or device 920, 925, 930, such as todistribute control of the extended reality environment among multipledevices. In addition, one or more steps of process 1100 may beincorporated into or combined with one or more steps of any otherprocess or embodiment (e.g., process 1000 of FIG. 10, process 1200 ofFIG. 12, process 1300 of FIG. 13, process 1400 of FIG. 14, process 1500of FIG. 15).

At 1102, I/O circuitry (e.g., I/O circuitry of head-mounted display 202)may receive a user selection to enter an XR environment (e.g., a VR, AR,or MR environment). Such user selection may be received via input (e.g.,voice command or touchscreen).

At 1104, control circuitry (e.g., control circuitry 804 of FIG. 8 and/orcontrol circuitry of head-mounted display 202 of FIG. 2) may, inresponse to receiving the user selection, generate for display (e.g.,via head-mounted device 202 of FIG. 2) the XR environment includingmultiple objects (e.g., car object 204 of FIG. 2 and airplane object 208of FIG. 2).

At 1106, I/O circuitry (e.g., I/O circuitry of head-mounted display 102)may receive input from a sensor (e.g., a camera). The sensor may detectvarious attributes of the eye of the user (e.g., eyelid motion, gaze).

At 1108, the control circuitry (e.g., control circuitry 804 of FIG. 8and/or control circuitry of head-mounted display 202 of FIG. 1) mayidentify objects (e.g., car object 204 of FIG. 2 and airplane object 208of FIG. 2) in a field of view of the user, based on the received input.In some embodiments, the control circuitry may generate a data structurefor the field of view, including object identifiers associated withvirtual objects in the field of view, and such data structure mayinclude coordinates representing the position of the field of view inthe XR environment. The control circuitry may determine the presentfield of view based on the data structure and/or images captured by theXR device 202, and identify objects in the field of view of the user.

At 1110, the control circuitry may detect eyelid motion (e.g., an amountof eyelid motion, a blink pattern, eyelid levels, etc.) of the userbased on the received input from the sensor (e.g., a camera). In someembodiments, such eyelid motion may be distinguished from involuntaryblinks, to avoid interpreting such involuntary blinks of the user as acommand. For example, the system may disregard blink patterns where theeye remains closed for less than a predetermined amount of time (e.g.,0.5 seconds).

At 1112, the control circuitry may determine whether the eyelid motionis associated with a first object (e.g., car object 204 of FIG. 2). Inthe example of FIG. 2, the control circuitry may determine whether thedetected eyelid motion matches the eyelid motion identifier “Blink onceto modify details of car” (e.g., stored in table 316 of FIG. 3).

At 1114, upon determining that the detected eyelid motion matches theeyelid motion identifier for the first object (e.g., car object 204 ofFIG. 2), the control circuitry may regenerate for display such objectwith a modified level of detail. For example, the control circuitry mayregenerate for display such object in a higher resolution than initiallypresented upon detecting a particular blink pattern, or at a larger sizethan initially presented upon detecting a particular blink pattern.

At 1116, if the control circuitry determines that the eyelid motion isnot associated with a first object (e.g., car object 204 of FIG. 2), thecontrol circuitry may determine whether the eyelid motion is associatedwith a another object (e.g., airplane object 208 of FIG. 2). In theexample of FIG. 2, the control circuitry may determine whether thedetected eyelid motion matches the eyelid motion identifier “Blink twiceto modify details of airplane” (e.g., stored in table 316 of FIG. 3).

At 1118, upon determining that the detected eyelid motion matches theeyelid motion identifier for the second object (e.g., airplane object208 of FIG. 2), the control circuitry may regenerate for display suchobject with a modified level of detail. For example, the controlcircuitry may regenerate for display the object in a higher resolutionthan initially presented upon detecting a particular blink pattern, orat a larger size than initially presented upon detecting a particularblink pattern. Thus, the user may selectively cause his or her gaze orfocus to shift in the XR environment by actuating his or her eyelids.

FIG. 12 is a flowchart of a detailed illustrative process for performingan action on an object in an XR environment, in accordance with someembodiments of this disclosure. For example, process 1200 may beexecuted by network device 800 (e.g., via control circuitry 804), and/orcontrol circuitry of the server 900, and/or control circuitry of device920, 925, 930, as instructed by one or more programs ofcomputer-executable instructions that may be implemented on networkdevice 800, and/or server 900, and/or device 920, 925, 930, such as todistribute control of the extended reality environment among multipledevices. In addition, one or more steps of process 1200 may beincorporated into or combined with one or more steps of any otherprocess or embodiment (e.g., process 1000 of FIG. 10, process 1100 ofFIG. 11, process 1300 of FIG. 13, process 1400 of FIG. 14, process 1500of FIG. 15).

At 1202, I/O circuitry (e.g., I/O circuitry of head-mounted display 302)may receive a user selection to enter an XR environment (e.g., a VR, AR,or MR environment). Such user selection may be received via input (e.g.,voice command or touch screen).

At 1204, control circuitry (e.g., control circuitry 804 of FIG. 8 and/orcontrol circuitry of head-mounted display 302 of FIG. 3) may, inresponse to receiving the user selection, generate for display (e.g.,via head-mounted device 302 of FIG. 3) the XR environment including anobject (e.g., book object 304 of FIG. 3). In some embodiments, thecontrol circuitry may generate a plurality of objects (e.g., book object304, lamp object 310, desk 312, chair 314 of FIG. 3).

At 1206, the control circuitry may store in memory (e.g., storage 808 ofFIG. 8 and/or storage of head-mounted display 302 of FIG. 3), or accessin memory a table (e.g., table 316 of FIG. 3) of eyelid motionidentifiers and corresponding actions performable on object.

At 1208, the control circuitry may detect eyelid motion (e.g., an amountof eyelid motion, a blink pattern, eyelid levels, etc.) of the userbased on the received input from a sensor (e.g., a camera). In someembodiments, such eyelid motion may be distinguished from involuntaryblinks, to avoid interpreting such involuntary blink of the user as acommand. For example, the system may disregard blink patterns where theeye remains closed for less than a predetermined amount of time (e.g.,0.5 seconds).

At 1210, the control circuitry determines whether the detected eyelidmotion matches any of the identifiers in the table (e.g., table 316 ofFIG. 3). In some embodiments, the control circuitry may analyze thesensor output and compare such output to the stored identifiers (e.g.,predetermined number of blinks, blink patterns, amount of eyelid motion,eyelid levels, etc.). In some embodiments, the stored identifiers mayinclude eyelid motions in combination with voice commands or otherinputs.

At 1212, the control circuitry determines whether the detected eyelidmotion matches a stored eyelid motion identifier. In some embodiments,the control circuitry may compute a match score, and may determine thatthere is a match between the detected eyelid motion and the storedeyelid motion identifier when the computed match score exceeds apredetermined threshold.

At 1214, upon determining that the detected eyelid motion matches astored eyelid motion identifier, the control circuitry generates fordisplay an updated version of the XR environment based on actioncorresponding to matched eyelid motion. In the example of FIG. 3, theupdated version of XR environment 300 includes book object 304 having aflipped page (e.g., in the environment 300 illustrated in the bottomportion of FIG. 3, as compared to the environment 300 illustrated in thetop portion of FIG. 3, in which book object 304 is closed).

FIG. 13 is a flowchart of a detailed illustrative process for receivinga voice command while an indicator is in a vicinity of an object in anXR environment, in accordance with some embodiments of this disclosure.For example, process 1300 may be executed by network device 800 (e.g.,via control circuitry 804), and/or control circuitry of the server 900,and/or control circuitry of device 920, 925, 930, as instructed by oneor more programs of computer-executable instructions that may beimplemented on network device 800, and/or server 900, and/or device 920,925, 930, such as to distribute control of the extended realityenvironment among multiple devices. In addition, one or more steps ofprocess 1300 may be incorporated into or combined with one or more stepsof any other process or embodiment (e.g., process 1000 of FIG. 10,process 1100 of FIG. 11, process 1200 of FIG. 12, process 1400 of FIG.14, process 1500 of FIG. 15).

At 1302, I/O circuitry (e.g., I/O circuitry of head-mounted display 402of FIGS. 4A-4B) may receive a user selection to enter an XR environment(e.g., a VR, AR, or MR environment). Such user selection may be receivedvia input (e.g., voice command or touch screen).

At 1304, control circuitry (e.g., control circuitry 804 of FIG. 8 and/orcontrol circuitry of head-mounted display 402 of FIG. 4A) may, inresponse to receiving the user selection, generate for display (e.g.,via head-mounted device 402 of FIG. 4) the XR environment including anobject (e.g., object 408 of FIG. 4). In some embodiments, the controlcircuitry may generate a plurality of objects (e.g., objects 408 and 410of FIG. 4).

At 1306, the control circuitry may detect the gaze of the user. Forexample, a sensor (e.g., a camera) may be used to track retinal movementof the user, and such retinal movement of the user may be plotted to thedisplay of the XR environment (e.g., environment 400 of FIGS. 4A-4B).

At 1308, the control circuitry may determine whether the gaze of theuser has shifted to a second portion of the XR environment including anobject (e.g., object 410 of FIGS. 4A-4B). In some embodiments, the gazeof the user may have shifted from a first portion of the display inwhich no object is included. Alternatively, such first portion of thedisplay may include an object (e.g., object 408 of FIGS. 4A-4B).

At 1310, in response to determining that the gaze of the user hasshifted to such second portion of the XR environment including an object(e.g., object 410 of FIGS. 4A-4B), the control circuitry may generatefor display an indicator (e.g., indicator 406 of FIG. 4) of the shift ingaze. For example, such indicator enables a user to confirm that thesystem is accurately tracking his or her gaze.

At 1312, in response to determining that the gaze of the user has notshifted to such second portion of XR environment, the control circuitrymay determine that the gaze of the user has not shifted, or has shiftedto a different portion of the display, and may generate for display theindicator (e.g., indicator 406 of FIG. 4) at the portion of the displayat which the gaze of the user is directed.

At 1314, I/O circuitry (e.g., I/O circuitry of head-mounted display 402of FIGS. 4A-4B) may receive a voice command while the indicator (e.g.,indicator 406 of FIG. 4) is in a vicinity of the object of interest(e.g., object 408 of FIG. 4A or object 410 of FIG. 4B). In someembodiments, control circuitry (e.g., control circuitry 804 of FIG. 8and/or control circuitry of head-mounted display 402 of FIG. 4A) maydetermine whether the voice command relates to such object of interest.In the example of FIG. 4A, the control circuitry may extract keywordsfrom the voice command of “Play ‘The Dark Knight’” and compare theextracted keywords to metadata associated with the object (e.g., object408) to which the gaze of the user is directed, to determine whether thevoice command relates to the object of interest.

At 1316, the control circuitry may execute the action corresponding tothe voice command related to the object. For example, in the example ofFIG. 4B, object 410, an identifier for the media content item “AmericanPsycho,” is in a vicinity of indicators 406, and upon receiving thevoice command 405 (“Play ‘American Psycho’”), presentation of such mediaasset to the user may commence.

FIG. 14 is a flowchart of a detailed illustrative process for varying anopacity of indicators in an XR environment, in accordance with someembodiments of this disclosure. It should be noted that process 1400 orany step thereof could be performed on, or provided by, any of thedevices shown in FIGS. 8-9. For example, process 1400 may be executed bynetwork device 800 (e.g., via control circuitry 804), and/or controlcircuitry of the server 900, and/or control circuitry of device 920,925, 930, as instructed by one or more programs of computer-executableinstructions that may be implemented on network device 800, and/orserver 900, and/or device 920, 925, 930, such as to distribute controlof the extended reality environment among multiple devices. In addition,one or more steps of process 1400 may be incorporated into or combinedwith one or more steps of any other process or embodiment (e.g., process1000 of FIG. 10, process 1100 of FIG. 11, process 1200 of FIG. 12,process 1300 of FIG. 13, process 1500 of FIG. 15).

At 1402, I/O circuitry (e.g., I/O circuitry of head-mounted display 502of FIG. 5) may receive a user selection to enter an XR environment(e.g., a VR, AR, or MR environment). Such user selection may be receivedvia input (e.g., voice command or touch screen).

At 1404, control circuitry (e.g., control circuitry 804 of FIG. 8 and/orcontrol circuitry of head-mounted display 502 of FIG. 5) may, inresponse to receiving the user selection, generate for display (e.g.,via head-mounted device 502 of FIG. 5) the XR environment including anobject (e.g., object 508 of FIG. 5). In some embodiments, the controlcircuitry may generate a plurality of objects (e.g., objects 508 and 510of FIG. 5).

At 1406, the control circuitry may detect the gaze of the user. Forexample, a sensor (e.g., a camera) may be used to track retinal movementof the user, and such retinal movement of the user may be plotted to thedisplay of the XR environment (e.g., environment 500 of FIG. 5). Thecontrol circuitry may determine whether the gaze of the user is directedto a portion of the XR environment including an object (e.g., object 508of FIG. 5).

At 1410, upon determining that the gaze of the user is not directed to aportion of the XR environment including the object (e.g., object 508 ofFIG. 5), the control circuitry may determine whether the gaze of theuser is directed to a different portion of the XR environment includinga different object (e.g., object 510 of FIG. 5).

At 1410, upon determining that the gaze of the user is directed to aportion of the XR environment including an object (e.g., object 508 ofFIG. 5), the control circuitry may generate for display within the XRenvironment opacity-based indicators (e.g., indicators 506 of FIG. 6) invicinity of such portion of the XR environment. On the other hand, upondetermining that the gaze of the user is directed to a different portionof the XR environment including a different object (e.g., object 510 ofFIG. 5), the control circuitry may generate for display within XRenvironment opacity-based indicators (e.g., indicators 506 of FIG. 6) inthe vicinity of such different portion of the XR environment.

At 1412, the control circuitry may determine a boundary of the object ofinterest (e.g., object 508 in FIG. 5). For example, the controlcircuitry may perform edge detection techniques and/or pixel comparisonto determine the boundary of the object.

At 1414, the control circuitry varies opacity of at least oneopacity-based indicator (e.g., indicator 506 of FIG. 5) based on anidentified boundary of the object. In some embodiments, the opacity ofindicators overlapping the boundary of the object of interest is varied.For example, the opacity of indicators overlapping the boundary of theobject of interest may be decreased, to avoid obscuring the object ofinterest (FIG. 6), or may be increased, to emphasize that the gaze ofthe user is directed to the object of interest (FIG. 5).

FIG. 15 is a flowchart of a detailed illustrative process for presentingadditional information related to an item in an XR environment, inaccordance with some embodiments of this disclosure. It should be notedthat process 1500 or any step thereof could be performed on, or providedby, any of the devices shown in FIGS. 8-9. For example, process 1500 maybe executed by network device 800 (e.g., via control circuitry 804),and/or control circuitry of the server 900, and/or control circuitry ofdevice 920, 925, 930, as instructed by one or more programs ofcomputer-executable instructions that may be implemented on networkdevice 800, and/or server 900, and/or device 920, 925, 930, such as todistribute control of the extended reality environment among multipledevices. In addition, one or more steps of process 1500 may beincorporated into or combined with one or more steps of any otherprocess or embodiment (e.g., process 1000 of FIG. 10, process 1100 ofFIG. 11, process 1200 of FIG. 12, process 1300 of FIG. 13, process 1400of FIG. 14).

At 1502, I/O circuitry (e.g., I/O circuitry of head-mounted display 702of FIG. 7) may receive a user selection to enter an XR environment(e.g., a VR, AR, or MR environment). Such user selection may be receivedvia input (e.g., voice command or touchscreen).

At 1504, control circuitry (e.g., control circuitry 804 of FIG. 8 and/orcontrol circuitry of head-mounted display 702 of FIG. 7) may, inresponse to receiving the user selection, generate for display (e.g.,via head-mounted device 702 of FIG. 7) the XR environment including anobject (e.g., object 708 of FIG. 7). In some embodiments, the controlcircuitry may generate a plurality of objects (e.g., objects 702, 704,708 of FIG. 7).

At 1506, control circuitry (e.g., control circuitry 804 of FIG. 8 and/orcontrol circuitry of head-mounted display 702 of FIG. 7) may detect eyemotion of the user (e.g., using a sensor).

At 1508, the control circuitry may determine whether an object is in afield of view of the user for at least a predetermined period of time.In some embodiments, the control circuitry may generate a data structurefor the field of view, including object identifiers associated withvirtual objects in the field of view, and such data structure mayinclude coordinates representing the position of the field of view inthe VR environment. The control circuitry may determine the presentfield of view based on the data structure and/or images captured byhead-mounted display 702, and identify objects in the field of view ofthe user. In certain embodiments, the control circuitry may additionallydetect whether a gaze of the user is directed at the object within thefield of view.

At 1510, the control circuitry, in response to determining that theobject is in the field of view for at least the predetermined period oftime (e.g., five seconds), the control circuitry generates for displayin the XR environment one or more items (e.g., item 714 of FIG. 7)related to the object (e.g., object 708 of FIG. 7). In some embodiments,in determining whether the object is in the field of view for at leastthe predetermined period of time, the control circuitry may determinewhether the object remains in the field of view of the user for acontinuous (or non-continuous) period of time during the XR session.

The processes discussed above are intended to be illustrative and notlimiting. One skilled in the art would appreciate that the steps of theprocesses discussed herein may be omitted, modified, combined, and/orrearranged, and any additional steps may be performed without departingfrom the scope of the invention. More generally, the above disclosure ismeant to be exemplary and not limiting. Only the claims that follow aremeant to set bounds as to what the present invention includes.Furthermore, it should be noted that the features and limitationsdescribed in any one embodiment may be applied to any other embodimentherein, and flowcharts or examples relating to one embodiment may becombined with any other embodiment in a suitable manner, done indifferent orders, or done in parallel. In addition, the systems andmethods described herein may be performed in real time. It should alsobe noted that the systems and/or methods described above may be appliedto, or used in accordance with, other systems and/or methods.

What is claimed is:
 1. A method for extended reality environmentinteraction, comprising: generating for display an extended realityenvironment including an object; detecting an eye motion of a user;determining, based on the detecting, whether the object is in a field ofview of the user for at least a first predetermined period of time; inresponse to determining that the object is in the field of view of theuser for at least the first predetermined period of time, generating fordisplay in the extended reality environment one or more items related tothe object; determining that at least a second predetermined period oftime has elapsed from commencing the display of the one or more itemsrelated to the object without the object being in the field of view ofthe user; and ceasing display of the one or more items related to theobject in response to determining that the second predetermined periodof time has elapsed from commencing the display of the one or more itemsrelated to the object without the object being in the field of view ofthe user.
 2. The method of claim 1, wherein the one or more itemsrelated to the object comprise textual information, one or more images,video, or any combination thereof.
 3. The method of claim 1, wherein:determining that at least the second predetermined period of time haselapsed from commencing the display of the one or more items related tothe object without the object being in the field of view of the userfurther comprises determining that the object has not been in the fieldof view of the user for at least the first predetermined period of time;and ceasing the display of the one or more items related to the objectin response to determining that the second predetermined period of timehas elapsed from commencing the display of the one or more items relatedto the object without the object being in the field of view of the useris performed further in response to determining that the object has notbeen in the field of view of the user for at least the firstpredetermined period of time.
 4. The method of claim 1, whereindetecting the eye motion comprises monitoring an eyelid motion.
 5. Themethod of claim 1, wherein: generating for display in the extendedreality environment the one or more items related to the object isperformed in response to determining that the object is in the field ofview of the user, and in a gaze of the user, for at least the firstpredetermined period of time.
 6. The method of claim 1, whereindetermining, based on the detecting, whether the object is in the fieldof view of the user for the first predetermined period of time comprisesdetermining that the object is continuously in the field of view of theuser for the first predetermined period of time during an extendedreality session.
 7. The method of claim 1, wherein determining, based onthe detecting, whether the object is in the field of view of the userfor the first predetermined period of time comprises determining thatthe object is non-continuously in the field of view of the user for thefirst predetermined period of time during an extended reality session.8. The method of claim 1, further comprising: determining a new objectis in the field of view of the user for at least the first predeterminedtime; and in response to determining that the new object is in the fieldof view of the user for at least the first predetermined period of time,generating for display in the extended reality environment one or moreitems related to the new object, while continuing to generate fordisplay in the extended reality environment the one or more itemsrelated to the object.
 9. The method of claim 1, further comprising:permitting modification of an amount of time for the first predeterminedperiod of time.
 10. The method of claim 1, wherein the extended realityenvironment is presented via an extended reality head-mounted device.11. A system for extended reality environment interaction, comprising: adisplay; control circuitry configured to: generate for display anextended reality environment including an object, wherein the object ispresented at the display; detect an eye motion of a user; determine,based on the detecting, whether the object is in a field of view of theuser for at least a first predetermined period of time; in response todetermining that the object is in the field of view of the user for atleast the first predetermined period of time, generate for display inthe extended reality environment one or more items related to theobject; determine that at least a second predetermined period of timehas elapsed from commencing the display of the one or more items relatedto the object without the object being in the field of view of the user;and cease display of the one or more items related to the object inresponse to determining that the second predetermined period of time haselapsed from commencing the display of the one or more items related tothe object without the object being in the field of view of the user.12. The system of claim 11, wherein the one or more items related to theobject comprise textual information, one or more images, video, or anycombination thereof.
 13. The system of claim 11, wherein the controlcircuitry is further configured to: determine that at least the secondpredetermined period of time has elapsed from commencing the display ofthe one or more items related to the object without the object being inthe field of view of the user by determining that the object has notbeen in the field of view of the user for at least the firstpredetermined period of time; and cease the display of the one or moreitems related to the object in response to determining that the secondpredetermined period of time has elapsed from commencing the display ofthe one or more items related to the object without the object being inthe field of view of the user and further in response to determiningthat the object has not been in the field of view of the user for atleast the first predetermined period of time.
 14. The system of claim11, wherein the control circuitry is further configured to, in detectingthe eye motion, monitor an eyelid motion.
 15. The system of claim 11,wherein the control circuitry is further configured to generate fordisplay in the extended reality environment the one or more itemsrelated to the object in response to determining that the object is inthe field of view of the user, and in a gaze of the user, for at leastthe first predetermined period of time.
 16. The system of claim 11,wherein the control circuitry is further configured to, in determiningwhether the object is in the field of view of the user for the firstpredetermined period of time, determine that the object is continuouslyin the field of view of the user for the first predetermined period oftime during an extended reality session.
 17. The system of claim 11,wherein the control circuitry is further configured to, in determiningwhether the object is in the field of view of the user for the firstpredetermined period of time, determine that the object isnon-continuously in the field of view of the user for the firstpredetermined period of time during an extended reality session.
 18. Thesystem of claim 11, wherein the control circuitry is further configuredto: determine a new object is in the field of view of the user for atleast the first predetermined time; and in response to determining thatthe new object is in the field of view of the user for at least thefirst predetermined period of time, generate for display in the extendedreality environment one or more items related to the new object, whilecontinuing to generate for display in the extended reality environmentthe one or more items related to the object.
 19. The system of claim 11,wherein the control circuitry is further configured to: permittingmodification of an amount of time for the first predetermined period oftime.
 20. The system of claim 11, wherein the extended realityenvironment is presented via an extended reality head-mounted device.